{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# 语言翻译\n",
    "\n",
    "在此项目中，你将了解神经网络机器翻译这一领域。你将用由英语和法语语句组成的数据集，训练一个序列到序列模型（sequence to sequence model），该模型能够将新的英语句子翻译成法语。\n",
    "\n",
    "## 获取数据\n",
    "\n",
    "因为将整个英语语言内容翻译成法语需要大量训练时间，所以我们提供了一小部分的英语语料库。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "source_path = 'data/small_vocab_en'\n",
    "target_path = 'data/small_vocab_fr'\n",
    "source_text = helper.load_data(source_path)\n",
    "target_text = helper.load_data(target_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 探索数据\n",
    "\n",
    "研究 view_sentence_range，查看并熟悉该数据的不同部分。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "view_sentence_range = (0, 10)\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import numpy as np\n",
    "\n",
    "print('Dataset Stats')\n",
    "print('Roughly the number of unique words: {}'.format(len({word: None for word in source_text.split()})))\n",
    "\n",
    "sentences = source_text.split('\\n')\n",
    "word_counts = [len(sentence.split()) for sentence in sentences]\n",
    "print('Number of sentences: {}'.format(len(sentences)))\n",
    "print('Average number of words in a sentence: {}'.format(np.average(word_counts)))\n",
    "\n",
    "print()\n",
    "print('English sentences {} to {}:'.format(*view_sentence_range))\n",
    "print('\\n'.join(source_text.split('\\n')[view_sentence_range[0]:view_sentence_range[1]]))\n",
    "print()\n",
    "print('French sentences {} to {}:'.format(*view_sentence_range))\n",
    "print('\\n'.join(target_text.split('\\n')[view_sentence_range[0]:view_sentence_range[1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 实现预处理函数\n",
    "\n",
    "### 文本到单词 id\n",
    "\n",
    "和之前的 RNN 一样，你必须首先将文本转换为数字，这样计算机才能读懂。在函数 `text_to_ids()` 中，你需要将单词中的 `source_text` 和 `target_text` 转为 id。但是，你需要在 `target_text` 中每个句子的末尾，添加 `<EOS>` 单词 id。这样可以帮助神经网络预测句子应该在什么地方结束。\n",
    "\n",
    "\n",
    "你可以通过以下代码获取  `<EOS> ` 单词ID：\n",
    "\n",
    "```python\n",
    "target_vocab_to_int['<EOS>']\n",
    "```\n",
    "\n",
    "你可以使用 `source_vocab_to_int` 和 `target_vocab_to_int` 获得其他单词 id。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def text_to_ids(source_text, target_text, source_vocab_to_int, target_vocab_to_int):\n",
    "    \"\"\"\n",
    "    Convert source and target text to proper word ids\n",
    "    :param source_text: String that contains all the source text.\n",
    "    :param target_text: String that contains all the target text.\n",
    "    :param source_vocab_to_int: Dictionary to go from the source words to an id\n",
    "    :param target_vocab_to_int: Dictionary to go from the target words to an id\n",
    "    :return: A tuple of lists (source_id_text, target_id_text)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    return None, None\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_text_to_ids(text_to_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 预处理所有数据并保存\n",
    "\n",
    "运行以下代码单元，预处理所有数据，并保存到文件中。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "helper.preprocess_and_save_data(source_path, target_path, text_to_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# 检查点\n",
    "\n",
    "这是你的第一个检查点。如果你什么时候决定再回到该记事本，或需要重新启动该记事本，可以从这里继续。预处理的数据已保存到磁盘上。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import numpy as np\n",
    "import helper\n",
    "\n",
    "(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 检查 TensorFlow 版本，确认可访问 GPU\n",
    "\n",
    "这一检查步骤，可以确保你使用的是正确版本的 TensorFlow，并且能够访问 GPU。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "from distutils.version import LooseVersion\n",
    "import warnings\n",
    "import tensorflow as tf\n",
    "\n",
    "# Check TensorFlow Version\n",
    "assert LooseVersion(tf.__version__) in [LooseVersion('1.0.0'), LooseVersion('1.0.1')], 'This project requires TensorFlow version 1.0  You are using {}'.format(tf.__version__)\n",
    "print('TensorFlow Version: {}'.format(tf.__version__))\n",
    "\n",
    "# Check for a GPU\n",
    "if not tf.test.gpu_device_name():\n",
    "    warnings.warn('No GPU found. Please use a GPU to train your neural network.')\n",
    "else:\n",
    "    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 构建神经网络\n",
    "\n",
    "你将通过实现以下函数，构建出要构建一个序列到序列模型所需的组件：\n",
    "\n",
    "- `model_inputs`\n",
    "- `process_decoding_input`\n",
    "- `encoding_layer`\n",
    "- `decoding_layer_train`\n",
    "- `decoding_layer_infer`\n",
    "- `decoding_layer`\n",
    "- `seq2seq_model`\n",
    "\n",
    "### 输入\n",
    "\n",
    "实现 `model_inputs()` 函数，为神经网络创建 TF 占位符。该函数应该创建以下占位符：\n",
    "\n",
    "- 名为 “input” 的输入文本占位符，并使用 TF Placeholder 名称参数（等级（Rank）为 2）。\n",
    "- 目标占位符（等级为 2）。\n",
    "- 学习速率占位符（等级为 0）。\n",
    "- 名为 “keep_prob” 的保留率占位符，并使用 TF Placeholder 名称参数（等级为 0）。\n",
    "\n",
    "在以下元祖（tuple）中返回占位符：（输入、目标、学习速率、保留率）\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def model_inputs():\n",
    "    \"\"\"\n",
    "    Create TF Placeholders for input, targets, and learning rate.\n",
    "    :return: Tuple (input, targets, learning rate, keep probability)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    return None, None, None, None\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_model_inputs(model_inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 处理解码输入\n",
    "\n",
    "使用 TensorFlow 实现 `process_decoding_input`，以便删掉 `target_data` 中每个批次的最后一个单词 ID，并将 GO ID 放到每个批次的开头。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def process_decoding_input(target_data, target_vocab_to_int, batch_size):\n",
    "    \"\"\"\n",
    "    Preprocess target data for dencoding\n",
    "    :param target_data: Target Placehoder\n",
    "    :param target_vocab_to_int: Dictionary to go from the target words to an id\n",
    "    :param batch_size: Batch Size\n",
    "    :return: Preprocessed target data\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    return None\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_process_decoding_input(process_decoding_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 编码\n",
    "\n",
    "实现 `encoding_layer()`，以使用 [`tf.nn.dynamic_rnn()`](https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn) 创建编码器 RNN 层级。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob):\n",
    "    \"\"\"\n",
    "    Create encoding layer\n",
    "    :param rnn_inputs: Inputs for the RNN\n",
    "    :param rnn_size: RNN Size\n",
    "    :param num_layers: Number of layers\n",
    "    :param keep_prob: Dropout keep probability\n",
    "    :return: RNN state\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    return None\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_encoding_layer(encoding_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 解码 - 训练\n",
    "\n",
    "使用 [`tf.contrib.seq2seq.simple_decoder_fn_train()`](https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/contrib/seq2seq/simple_decoder_fn_train) 和 [`tf.contrib.seq2seq.dynamic_rnn_decoder()`](https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/contrib/seq2seq/dynamic_rnn_decoder) 创建训练分对数（training logits）。将 `output_fn` 应用到 [`tf.contrib.seq2seq.dynamic_rnn_decoder()`](https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/contrib/seq2seq/dynamic_rnn_decoder) 输出上。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def decoding_layer_train(encoder_state, dec_cell, dec_embed_input, sequence_length, decoding_scope,\n",
    "                         output_fn, keep_prob):\n",
    "    \"\"\"\n",
    "    Create a decoding layer for training\n",
    "    :param encoder_state: Encoder State\n",
    "    :param dec_cell: Decoder RNN Cell\n",
    "    :param dec_embed_input: Decoder embedded input\n",
    "    :param sequence_length: Sequence Length\n",
    "    :param decoding_scope: TenorFlow Variable Scope for decoding\n",
    "    :param output_fn: Function to apply the output layer\n",
    "    :param keep_prob: Dropout keep probability\n",
    "    :return: Train Logits\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    return None\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_decoding_layer_train(decoding_layer_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 解码 - 推论\n",
    "\n",
    "使用 [`tf.contrib.seq2seq.simple_decoder_fn_inference()`](https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/contrib/seq2seq/simple_decoder_fn_inference) 和 [`tf.contrib.seq2seq.dynamic_rnn_decoder()`](https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/contrib/seq2seq/dynamic_rnn_decoder) 创建推论分对数（inference logits）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id,\n",
    "                         maximum_length, vocab_size, decoding_scope, output_fn, keep_prob):\n",
    "    \"\"\"\n",
    "    Create a decoding layer for inference\n",
    "    :param encoder_state: Encoder state\n",
    "    :param dec_cell: Decoder RNN Cell\n",
    "    :param dec_embeddings: Decoder embeddings\n",
    "    :param start_of_sequence_id: GO ID\n",
    "    :param end_of_sequence_id: EOS Id\n",
    "    :param maximum_length: The maximum allowed time steps to decode\n",
    "    :param vocab_size: Size of vocabulary\n",
    "    :param decoding_scope: TensorFlow Variable Scope for decoding\n",
    "    :param output_fn: Function to apply the output layer\n",
    "    :param keep_prob: Dropout keep probability\n",
    "    :return: Inference Logits\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    return None\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_decoding_layer_infer(decoding_layer_infer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 构建解码层级\n",
    "\n",
    "实现 `decoding_layer()` 以创建解码器 RNN 层级。\n",
    "\n",
    "- 使用 `rnn_size` 和 `num_layers` 创建解码 RNN 单元。\n",
    "- 使用 [`lambda`](https://docs.python.org/3/tutorial/controlflow.html#lambda-expressions) 创建输出函数，将输入，也就是分对数转换为类分对数（class logits）。\n",
    "- 使用 `decoding_layer_train(encoder_state, dec_cell, dec_embed_input, sequence_length, decoding_scope, output_fn, keep_prob)` 函数获取训练分对数。\n",
    "- 使用 `decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id, maximum_length, vocab_size, decoding_scope, output_fn, keep_prob)` 函数获取推论分对数。\n",
    "\n",
    "注意：你将需要使用 [tf.variable_scope](https://www.tensorflow.org/api_docs/python/tf/variable_scope) 在训练和推论分对数间分享变量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def decoding_layer(dec_embed_input, dec_embeddings, encoder_state, vocab_size, sequence_length, rnn_size,\n",
    "                   num_layers, target_vocab_to_int, keep_prob):\n",
    "    \"\"\"\n",
    "    Create decoding layer\n",
    "    :param dec_embed_input: Decoder embedded input\n",
    "    :param dec_embeddings: Decoder embeddings\n",
    "    :param encoder_state: The encoded state\n",
    "    :param vocab_size: Size of vocabulary\n",
    "    :param sequence_length: Sequence Length\n",
    "    :param rnn_size: RNN Size\n",
    "    :param num_layers: Number of layers\n",
    "    :param target_vocab_to_int: Dictionary to go from the target words to an id\n",
    "    :param keep_prob: Dropout keep probability\n",
    "    :return: Tuple of (Training Logits, Inference Logits)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    return None, None\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_decoding_layer(decoding_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 构建神经网络\n",
    "\n",
    "应用你在上方实现的函数，以：\n",
    "\n",
    "- 向编码器的输入数据应用嵌入。\n",
    "- 使用 `encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob)` 编码输入。\n",
    "- 使用 `process_decoding_input(target_data, target_vocab_to_int, batch_size)` 函数处理目标数据。\n",
    "- 向解码器的目标数据应用嵌入。\n",
    "- 使用 `decoding_layer(dec_embed_input, dec_embeddings, encoder_state, vocab_size, sequence_length, rnn_size, num_layers, target_vocab_to_int, keep_prob)` 解码编码的输入数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def seq2seq_model(input_data, target_data, keep_prob, batch_size, sequence_length, source_vocab_size, target_vocab_size,\n",
    "                  enc_embedding_size, dec_embedding_size, rnn_size, num_layers, target_vocab_to_int):\n",
    "    \"\"\"\n",
    "    Build the Sequence-to-Sequence part of the neural network\n",
    "    :param input_data: Input placeholder\n",
    "    :param target_data: Target placeholder\n",
    "    :param keep_prob: Dropout keep probability placeholder\n",
    "    :param batch_size: Batch Size\n",
    "    :param sequence_length: Sequence Length\n",
    "    :param source_vocab_size: Source vocabulary size\n",
    "    :param target_vocab_size: Target vocabulary size\n",
    "    :param enc_embedding_size: Decoder embedding size\n",
    "    :param dec_embedding_size: Encoder embedding size\n",
    "    :param rnn_size: RNN Size\n",
    "    :param num_layers: Number of layers\n",
    "    :param target_vocab_to_int: Dictionary to go from the target words to an id\n",
    "    :return: Tuple of (Training Logits, Inference Logits)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    return None\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_seq2seq_model(seq2seq_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 训练神经网络\n",
    "\n",
    "### 超参数\n",
    "\n",
    "调试以下参数：\n",
    "\n",
    "- 将 `epochs` 设为 epoch 次数。\n",
    "- 将 `batch_size` 设为批次大小。\n",
    "- 将 `rnn_size` 设为 RNN 的大小。\n",
    "- 将 `num_layers` 设为层级数量。\n",
    "- 将 `encoding_embedding_size` 设为编码器嵌入大小。\n",
    "- 将 `decoding_embedding_size` 设为解码器嵌入大小\n",
    "- 将 `learning_rate` 设为训练速率。\n",
    "- 将 `keep_probability` 设为丢弃保留率（Dropout keep probability）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Number of Epochs\n",
    "epochs = None\n",
    "# Batch Size\n",
    "batch_size = None\n",
    "# RNN Size\n",
    "rnn_size = None\n",
    "# Number of Layers\n",
    "num_layers = None\n",
    "# Embedding Size\n",
    "encoding_embedding_size = None\n",
    "decoding_embedding_size = None\n",
    "# Learning Rate\n",
    "learning_rate = None\n",
    "# Dropout Keep Probability\n",
    "keep_probability = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 构建图表\n",
    "\n",
    "使用你实现的神经网络构建图表。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "save_path = 'checkpoints/dev'\n",
    "(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()\n",
    "max_source_sentence_length = max([len(sentence) for sentence in source_int_text])\n",
    "\n",
    "train_graph = tf.Graph()\n",
    "with train_graph.as_default():\n",
    "    input_data, targets, lr, keep_prob = model_inputs()\n",
    "    sequence_length = tf.placeholder_with_default(max_source_sentence_length, None, name='sequence_length')\n",
    "    input_shape = tf.shape(input_data)\n",
    "    \n",
    "    train_logits, inference_logits = seq2seq_model(\n",
    "        tf.reverse(input_data, [-1]), targets, keep_prob, batch_size, sequence_length, len(source_vocab_to_int), len(target_vocab_to_int),\n",
    "        encoding_embedding_size, decoding_embedding_size, rnn_size, num_layers, target_vocab_to_int)\n",
    "\n",
    "    tf.identity(inference_logits, 'logits')\n",
    "    with tf.name_scope(\"optimization\"):\n",
    "        # Loss function\n",
    "        cost = tf.contrib.seq2seq.sequence_loss(\n",
    "            train_logits,\n",
    "            targets,\n",
    "            tf.ones([input_shape[0], sequence_length]))\n",
    "\n",
    "        # Optimizer\n",
    "        optimizer = tf.train.AdamOptimizer(lr)\n",
    "\n",
    "        # Gradient Clipping\n",
    "        gradients = optimizer.compute_gradients(cost)\n",
    "        capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]\n",
    "        train_op = optimizer.apply_gradients(capped_gradients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 训练\n",
    "\n",
    "利用预处理的数据训练神经网络。如果很难获得低损失值，请访问我们的论坛，看看其他人是否遇到了相同的问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import time\n",
    "\n",
    "def get_accuracy(target, logits):\n",
    "    \"\"\"\n",
    "    Calculate accuracy\n",
    "    \"\"\"\n",
    "    max_seq = max(target.shape[1], logits.shape[1])\n",
    "    if max_seq - target.shape[1]:\n",
    "        target = np.pad(\n",
    "            target,\n",
    "            [(0,0),(0,max_seq - target.shape[1])],\n",
    "            'constant')\n",
    "    if max_seq - logits.shape[1]:\n",
    "        logits = np.pad(\n",
    "            logits,\n",
    "            [(0,0),(0,max_seq - logits.shape[1]), (0,0)],\n",
    "            'constant')\n",
    "\n",
    "    return np.mean(np.equal(target, np.argmax(logits, 2)))\n",
    "\n",
    "train_source = source_int_text[batch_size:]\n",
    "train_target = target_int_text[batch_size:]\n",
    "\n",
    "valid_source = helper.pad_sentence_batch(source_int_text[:batch_size])\n",
    "valid_target = helper.pad_sentence_batch(target_int_text[:batch_size])\n",
    "\n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "\n",
    "    for epoch_i in range(epochs):\n",
    "        for batch_i, (source_batch, target_batch) in enumerate(\n",
    "                helper.batch_data(train_source, train_target, batch_size)):\n",
    "            start_time = time.time()\n",
    "            \n",
    "            _, loss = sess.run(\n",
    "                [train_op, cost],\n",
    "                {input_data: source_batch,\n",
    "                 targets: target_batch,\n",
    "                 lr: learning_rate,\n",
    "                 sequence_length: target_batch.shape[1],\n",
    "                 keep_prob: keep_probability})\n",
    "            \n",
    "            batch_train_logits = sess.run(\n",
    "                inference_logits,\n",
    "                {input_data: source_batch, keep_prob: 1.0})\n",
    "            batch_valid_logits = sess.run(\n",
    "                inference_logits,\n",
    "                {input_data: valid_source, keep_prob: 1.0})\n",
    "                \n",
    "            train_acc = get_accuracy(target_batch, batch_train_logits)\n",
    "            valid_acc = get_accuracy(np.array(valid_target), batch_valid_logits)\n",
    "            end_time = time.time()\n",
    "            print('Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.3f}, Validation Accuracy: {:>6.3f}, Loss: {:>6.3f}'\n",
    "                  .format(epoch_i, batch_i, len(source_int_text) // batch_size, train_acc, valid_acc, loss))\n",
    "\n",
    "    # Save Model\n",
    "    saver = tf.train.Saver()\n",
    "    saver.save(sess, save_path)\n",
    "    print('Model Trained and Saved')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 保存参数\n",
    "\n",
    "保存 `batch_size` 和 `save_path` 参数以进行推论（for inference）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "# Save parameters for checkpoint\n",
    "helper.save_params(save_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# 检查点"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "_, (source_vocab_to_int, target_vocab_to_int), (source_int_to_vocab, target_int_to_vocab) = helper.load_preprocess()\n",
    "load_path = helper.load_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 句子到序列\n",
    "\n",
    "要向模型提供要翻译的句子，你首先需要预处理该句子。实现函数 `sentence_to_seq()` 以预处理新的句子。\n",
    "\n",
    "- 将句子转换为小写形式\n",
    "- 使用 `vocab_to_int` 将单词转换为 id\n",
    " - 如果单词不在词汇表中，将其转换为`<UNK>` 单词 id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def sentence_to_seq(sentence, vocab_to_int):\n",
    "    \"\"\"\n",
    "    Convert a sentence to a sequence of ids\n",
    "    :param sentence: String\n",
    "    :param vocab_to_int: Dictionary to go from the words to an id\n",
    "    :return: List of word ids\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    return None\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_sentence_to_seq(sentence_to_seq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 翻译\n",
    "\n",
    "将 `translate_sentence` 从英语翻译成法语。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "translate_sentence = 'he saw a old yellow truck .'\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "translate_sentence = sentence_to_seq(translate_sentence, source_vocab_to_int)\n",
    "\n",
    "loaded_graph = tf.Graph()\n",
    "with tf.Session(graph=loaded_graph) as sess:\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(load_path + '.meta')\n",
    "    loader.restore(sess, load_path)\n",
    "\n",
    "    input_data = loaded_graph.get_tensor_by_name('input:0')\n",
    "    logits = loaded_graph.get_tensor_by_name('logits:0')\n",
    "    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')\n",
    "\n",
    "    translate_logits = sess.run(logits, {input_data: [translate_sentence], keep_prob: 1.0})[0]\n",
    "\n",
    "print('Input')\n",
    "print('  Word Ids:      {}'.format([i for i in translate_sentence]))\n",
    "print('  English Words: {}'.format([source_int_to_vocab[i] for i in translate_sentence]))\n",
    "\n",
    "print('\\nPrediction')\n",
    "print('  Word Ids:      {}'.format([i for i in np.argmax(translate_logits, 1)]))\n",
    "print('  French Words: {}'.format([target_int_to_vocab[i] for i in np.argmax(translate_logits, 1)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 不完美的翻译\n",
    "\n",
    "你可能注意到了，某些句子的翻译质量比其他的要好。因为你使用的数据集只有 227 个英语单词，但实际生活中有数千个单词，只有使用这些单词的句子结果才会比较理想。对于此项目，不需要达到完美的翻译。但是，如果你想创建更好的翻译模型，则需要更好的数据。\n",
    "\n",
    "你可以使用 [WMT10 French-English corpus](http://www.statmt.org/wmt10/training-giga-fren.tar) 语料库训练模型。该数据集拥有更多的词汇，讨论的话题也更丰富。但是，训练时间要好多天的时间，所以确保你有 GPU 并且对于我们提供的数据集，你的神经网络性能很棒。提交此项目后，别忘了研究下 WMT10 语料库。\n",
    "\n",
    "\n",
    "## 提交项目\n",
    "\n",
    "提交项目时，确保先运行所有单元，然后再保存记事本。保存记事本文件为 “dlnd_language_translation.ipynb”，再通过菜单中的“文件” ->“下载为”将其另存为 HTML 格式。提交的项目文档中需包含“helper.py”和“problem_unittests.py”文件。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
